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SUMMARY 


The  Human  Effectiveness  Directorate  at  the  Air  Force  Research  Laboratory  (AFRL)  has  recently 
adopted  a  Sense-Assess- Augment  (S-A-A)  taxonomy  to  facilitate  the  planning,  development, 
execution,  and  reporting  of  their  human-centered  research  portfolio.  The  objective  of  this  taxonomy 
is  to  sense  individual  and  team  cognitive  state,  assess  the  state  relative  to  performance,  and  augment 
performance  to  optimize  mission  effectiveness.  This  taxonomy  is  being  applied  to  revolutionize 
improvements  in  human  performance  by  leveraging  the  integration  of  several  sensing  technologies 
coupled  with  multiple  assessment  approaches  to  provide  a  robust  understanding  of  the  causes  of 
operator  performance  decrements.  Given  a  better  understanding  of  the  causes  for  sub-optimal 
perfonnance,  targeted  augmentation  techniques  can  be  employed  to  improve  individual  or  team 
performance.  The  adopted  taxonomy  has  been  utilized  in  the  Human  Universal  Measurement  and 
Assessment  Network  (HUMAN)  laboratory  at  the  Air  Force  Research  Laboratory  to  examine 
bottlenecks  associated  with  controlling  multiple  Remotely  Piloted  Aircraft  (RPAs).  This  example  will 
be  used  to  illustrate  how  the  S-A-A  process  was  instantiated  and  utilized  to  identify  and  implement 
targeted  augmentations  that  improve  operator  perfonnance. 

INTRODUCTION 

In  2010,  the  Chief  Scientist  of  the  Air  Force  released  a  report  outlining  the  science  and  technology 
needs  in  the  2010-2030  time  frame  (Dahm,  2010).  A  key  finding  from  that  report  was  that  natural 
human  capacities  are  becoming  increasingly  mismatched  to  the  enormous  data  volumes,  processing 
capabilities,  and  decision  speeds  that  technologies  either  offer  or  demand.  Although  humans  today 
remain  more  capable  than  machines  for  many  tasks,  by  2030  it  is  hypothesized  that  machine 
capabilities  will  be  so  advanced  that  humans  will  become  the  weakest  component  in  a  wide  array  of 
systems  and  processes.  Humans  and  machines  will  need  to  become  far  more  integrated,  through 
improved  human-machine  interfaces,  the  use  of  autonomous  systems,  and  by  direct  augmentation  of 
human  perfonnance  in  order  to  facilitate  the  realization  of  these  increased  machine  capabilities. 

Developing  ways  of  using  science  and  technology  to  augment  human  performance  will  become 
increasingly  essential  for  gaining  the  benefits  that  many  existing,  planned,  and  future  technologies 
can  bring.  Significant  advances  and  early  implementations  are  possible  over  the  next  decade.  Such 
augmentation  techniques  are  a  further  means  for  increasing  human  efficiencies,  facilitating  reduced 
manpower  requirements  for  the  same  capabilities  or  even  increased  capabilities  with  given 
manpower. 

There  are  two  primary  questions  one  must  consider  before  implementing  human  perfonnance 
augmentation:  1)  when  to  provide  the  augmentation  and  2)  how  to  provide  the  augmentation.  The 
answers  to  these  questions  will  ensure  the  right  augmentation  technique  is  employed  at  the  right  time 
to  produce  the  desired  effects.  Recently,  the  Human  Effectiveness  Directorate  at  the  Air  Force 
Research  Laboratory  has  adopted  a  Sense-Assess-Augment  (S-A-A)  taxonomy  to  provide  answers  to 
these  questions.  The  objective  of  this  taxonomy  is  to  sense  individual  and  team  cognitive  state,  assess 
the  state  relative  to  perfonnance,  and  augment  perfonnance  to  optimize  mission  effectiveness 
(Figure  1).  The  Human  Effectiveness  Directorate  is  cunently  applying  this  taxonomy  to  conduct 
research  that  facilitates  improved  human  performance.  Human  performance  research,  therefore, 
leverages  the  integration  of  several  sensing  technologies  coupled  with  multiple  assessment 
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approaches  to  provide  a  robust  understanding  of  the  causes  of  operator  or  system  performance 
decrements.  Given  a  better  understanding  of  the  causes  for  sub-optimal  performance,  targeted 
augmentation  techniques  can  be  employed  to  improve  individual  or  team  performance. 


Sense  or  detect 
individual  or  team 
cognitive/functional  state 

Assess  the  state  relative 
to  performance 

Auament  performance 
to  optimize  mission 
effectiveness 

Figure  1.  Graphical 

epiction  of  the  Sense- Assess-Augment  (S-A-A)  taxonomy 

The  S-A-A  taxonomy  was  originally  presented  during  a  Human  Effectiveness  Directorate  Scientific 
Advisory  Board  review  (Galster,  2011).  The  taxonomy  served  to  facilitate  in  the  planning, 
development,  execution,  and  reporting  of  the  Applied  Neuroscience  human-centered  research 
portfolio.  In  the  following  year,  a  workshop  was  held  at  Arizona  State  University  where  participants 
from  academia,  industry,  and  military  sectors  convened  to  examine  the  S-A-A  taxonomy  and  its 
utility.  The  workshop  served  to  validate  the  taxonomy  and  included  recommendations  to  continue 
and  expand  its  use  as  a  common  way  to  describe  and  report  on  human-centric  research  and 
development  efforts.  An  article  describing  how  the  taxonomy  could  be  used  across  military  domains 
to  create  the  future  quantified  warrior  was  subsequently  published  by  the  Senior  Leaders  in  the 
Human  Effectiveness  Directorate  (Blackhurst,  Gresham,  &  Stone,  2012). 

Below,  tenets  of  the  S-A-A  taxonomy  are  described  followed  by  an  example  of  how  the  S-A-A 
taxonomy  is  currently  being  utilized  in  the  Human  Universal  Measurement  and  Assessment  Network 
(HUMAN)  laboratory  at  the  Air  Force  Research  Laboratory  at  Wright-Patterson  AFB,  OH. 


Sense 

A  snapshot  of  the  metrics  being  used  or  explored  at  AFRL  to  sense  cognitive  state  is  depicted  in 
Figure  2.  Below  each  metric  is  the  psychological  or  behavioral  construct  of  interest  typically 
associated  with  the  use  of  that  particular  metric.  Recently,  a  concerted  effort  has  been  on  integrating 
several  sensing  technologies  to  gather  a  more  robust  (reliable  and  proof  of  stability  over  time) 
“picture”  of  the  operator  cognitive  state. 

Of  course,  sensors  continue  to  evolve  and  mature  and  all  efforts  must  be  taken  to  integrate  and  test 
new  sensors  if  feasible.  It  is  also  advisable  to  plan  for  the  rapid  integration  of  new  sensing 
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technologies  to  the  greatest  extent  possible.  One  of  the  primary  objectives  in  the  sense  portion  of  the 
S-A-A  taxonomy  is  to  use  sensors  that  are  more  cost  effective,  disposable,  non-  or  minimally- 
invasive  and  provide  measures  that  are  accessible  in  near  real-time. 


Off  Body  On  Body 

( ^  tvi 


In  Body 


Eye  tracking/Oculometrics 

Galvanic  Skin  Response 
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Vigilance,  Arousal,  Trust, 
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Technical  Interface  Activity 

Stress/Anxiety 

Memory /Learning, 

Stress/Anxiety,  Cognitive  Load 
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Stress  Resilience 

Verbal  Protocol 

Cognitive  Load 

Catecholamines 

Context/Communication 
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Metrics  of  Performance 

Team  Synchronicity, 

Cytokines 

Reaction  time,  Cognitive  Load 

Arousal 

Inflammation/Stress 

V  J 

^  J 
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Figure  2.  Baseline  sensing  technologies 


Assess 

Assessment  techniques  can  vary  widely  and  are  typically  chosen  by  a  number  of  factors;  the  type  of 
data  being  collected,  the  frequency  and  accessibility  of  the  data  produced,  the  objectives  of  the 
evaluation,  and  expertise  of  personnel  involved  to  name  a  few.  One  major  differentiator  in  deciding 
what  type  of  assessment  technique  or  approach  to  utilize  is  whether  a  top-down,  model  driven 
approach  or  a  bottom-up  data  driven  approach  is  better  suited  to  a  particular  application.  Some 
examples  of  “standard”  assessment  techniques  include  the  use  of  nonlinear  classification  methods 
such  as  artificial  neural  networks,  kernel  based  support  vector  machines,  kernel  based  discriminant 
analysis,  and  the  use  of  manifold  learning  classifiers  (see  also,  Pereria,  Mitchell,  &  Botvinick,  2009; 
Christensen,  Estepp,  Wilson,  &  Russell,  2012).  There  will  always  be  benefits  and  risks  associated 
with  choosing  one  approach  over  another.  It  will  be  up  to  the  individual  utilizing  the  assessment 
method  to  determine  the  best  approach,  again  given  the  particular  application.  The  assessment  phase 
is  critical  in  determining  when  to  engage  the  augmentation.  A  key  component  in  answering  this 
question  is  completing  the  required  analysis  in  near  real  time  so  the  augmentation,  if  needed,  can  be 
deployed  before  the  degraded  operator  cognitive  state  causes  irreparable  consequences  in 
performance  that  may  result  in  mission  failure. 
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Augment 


Augmentation  approaches  span  the  use  of  neuropharmaceuticals,  biofeedback,  physical  training,  and 
brain  stimulation.  Techniques  such  as  biofeedback  and  brain  stimulation  were  originally  developed 
in  clinical  domains  to  treat  neurological  and  psychological  deficits,  however  their  potential  to 
enhance  cognitive  abilities  and  performance  in  healthy  individuals  have  recently  been  explored. 

These  techniques  have  the  potential  to  improve  multiple  facets  of  cognition  including  memory, 
alertness,  executive  function,  and  visual/aural  acuity.  More  controversial  techniques  such  as 
implants  or  even  genetic  modification  are  possible  augmentation  options.  While  such  methods  may 
not  be  culturally  acceptable  by  some,  potential  adversaries  may  be  willing  to  make  use  of  them 
without  reservation.  It  is  important  to  understand  all  augmentation  possibilities  and  outcomes. 
Developing  acceptable  ways  of  using  science  and  technology  to  augment  human  performance  will 
become  increasingly  essential  for  realizing  the  benefits  that  many  technologies  afford.  The  current 
technical  maturity  of  various  approaches  in  this  area  varies  widely,  but  significant  steps  to  advance 
and  develop  early  implementations  are  possible  now  and  over  the  next  decade. 

Application 

The  utilization  of  the  S-A-A  taxonomy  allows  researchers  and  practitioners  to  identify  specific 
bottlenecks  that  may  occur  during  operations.  The  correct  identification  of  the  individual’s  source  for 
sub-optimal  performance  will  drive  what  augmentation  method  should  be  utilized  to  provide 
increases  in  human  perfonnance.  There  are  efforts  currently  underway  to  utilize  the  S-A-A  taxonomy 
to  examine  the  use  of  contextually  relevant  augmentation  techniques  in  the  control  of  multiple 
remotely  piloted  aircraft  (RPAs).  One  such  augmentation  is  the  use  of  automation  to  ease  the  acute 
stress  and  cognitive  load  an  operator  may  experience  as  mission  demands  become  more  taxing  and 
complex. 

The  integration  of  automated  tools  into  highly  complex  systems  has  created  a  need  to  examine  the 
nature  of  the  interaction  between  humans  and  the  automated  tools  that  they  use.  The  examination  of 
human  performance  in  systems  that  utilize  automation  has  become  widespread.  Investigations  of 
human  interaction  with  automation  have  revealed  that  automation  does  not  always  function  in  the 
way  intended  by  designers  and,  moreover,  can  produce  deleterious  perfonnance  effects  (Bainbridge, 
1983;  Billings,  1997;  Billings  &  Woods,  1994;  Parasuraman  &  Riley,  1997;  Sarter  &  Woods,  1995; 
Woods,  1996).  Automation  can  change  the  nature  of  the  demands  on  the  operator  and  produce 
subsequent  changes  in  perfonnance  not  seen  when  automation  is  absent.  A  review  of  human 
perfonnance  costs  of  automation  list  possible  changes  in  the  mental  workload  for  the  operator,  an 
increase  in  the  monitoring  demands,  and  a  decrease  of  the  monitoring  efficiency  of  the  operator  as 
costs  contributing  to  poor  performance  (Parasuraman  &  Riley,  1997;  Sheridan,  2002).  A  reduction  in 
skill  due  to  lack  of  use  and  a  reduction  in  the  situation  awareness  of  the  operator  have  also  been 
identified  as  potential  costs  of  automation  usage.  The  issues  raised  above  suggest  that  automated 
tools  should  not  be  used  as  a  panacea;  rather  it  should  be  instantiated  deliberately  under  specified 
conditions  that  warrant  its  use. 

The  example  that  follows  is  utilizing  the  S-A-A  taxonomy  to  identify  when  an  operator  is  cognitively 
loaded,  what  consequences  there  are  to  performance  and  what  augmentation  technique  to  employ  if  a 
perfonnance  decrement  occurs  that  will  jeopardize  mission  assurance.  The  data  is  collected  and 
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analyzed  along  with  performance  data  to  compile  a  comprehensive  picture  of  drivers  that  may  affect 
operator  workload  and  decision  making  ability.  This  understanding  provides  opportunities  to 
implement  performance  augmentations  or  mitigations  to  alleviate  any  bottlenecks.  The  objective  of 
the  current  effort  is  to  provide  design  guidance  to  those  who  will  build  the  next  generation  RPA 
control  station.  The  goal  is  to  step  away  from  a  technically-centered  design  and  facilitate  a  design  that 
is  more  human-system  centric;  one  that  is  constantly  surveying  the  operator  cognitive  state, 
determining  if  performance  is  affected,  and  able  to  remediate  bottlenecks  that  may  endanger  mission 
performance. 

The  Human  Universal  Measurement  and  Assessment  Network  (HUMAN)  Laboratory 

The  Human  Universal  Measurement  and  Assessment  Network  Laboratory  (HUMAN)  Laboratory 
uses  a  comprehensive  measurement  and  assessment  approach  to  provide  performance  optimization 
and  mitigation  recommendations  that  are  based  on  a  thorough  understanding  of  the  complexities  that 
exist  in  contextually  rich  activities.  The  HUMAN  Laboratory  is  equipped  with  a  state-of-the-art 
multimodal  data  acquisition  system,  including  sensors  to  monitor  brain,  heart,  muscle-activity,  eye 
movement,  respiration,  galvanic  skin  response  and  other  body  signals.  These  data  and  perfonnance 
data  from  the  RPA  simulation  environment  are  fed  into  a  commercially  available  architecture, 
translated  to  an  XML-based  Human  Performance  Markup  Language  (HPML)  and  forwarded  to  a 
machine  learning-based  model  for  evaluation  (see  also  Durkee,  Geyer,  Pappada,  Ortiz,  &  Wiggins, 
2013;  Durkee,  Geyer,  Pappada,  Ortiz  &  Galster,  2013).  Additional  data  such  as  self  and  observer- 
based  measures  can  also  be  funneled  into  the  Universal  Data  Bus  for  consideration  in  the  evaluative 
process  (Figure  3). 

Data  collection  is  currently  underway  in  the  HUMAN  laboratory.  Initial  efforts  assured  that  the 
integrated  sensors  were  performing  as  planned,  that  data  was  being  published  to  the  Universal  Data 
Bus,  that  the  data  was  available  for  evaluation  from  the  machine  learning-based  model,  and  that  an 
initial  composite  real-time  workload  measure  was  being  produced.  The  next  step  in  this  process  is  to 
include  saliva-based  biomarkers  (dehydroepiandrosterone,  cortisol,  immunoglobulin  A,  and  alpha- 
amylase)  in  the  data  collection  process.  These  biomarkers  will  not  currently  contribute  to  the  real¬ 
time  operator  cognitive  state  assessment  but  they  will  play  a  critical  role  in  detennining  the 
relationship  between  changes  in  biochemistry  and  real-time  assessment  of  operator  cognitive 
workload  and  stress.  This  relationship  will  help  define  the  strategic  plan  for  future  sensor 
development. 
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Figure  3.  Representative  architecture  for  data  collection  in  the  HUMAN  laboratory.  From  Aptima,  Inc. 
DISCUSSION 


The  basic  tenets  of  the  S-A-A  taxonomy  were  described  with  an  example  illustrating  how  the 
taxonomy  is  being  applied  to  support  the  design  of  future  RPA  control  stations.  The  taxonomy  is 
currently  utilized  in  a  number  of  other  scientific  endeavors  including  the  examination  of  cyber 
operations,  pilot  performance,  threat  detection,  and  as  a  guiding  construct  to  improve  existing  testing, 
evaluation,  validation,  and  verification  methodologies. 
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